Translation assessment

ABSTRACT

A system and method for assessing translation quality are disclosed. In one example, a computer system receives data associated with a translation from a first text in a first natural language to a second text in a second natural language, wherein the data comprises a first value that is based on one or more semantic errors of the second text, a second value that is based on extraneous content in the second text, and a third value that is based on missing content in the second text; and determining by the computer system, based on the first value, the second value, and the third value, a score that quantifies translation quality of the second text with respect to the first text.

TECHNICAL FIELD

This disclosure relates to computer systems, and more particularly, toassessing the quality of translations.

BACKGROUND

Translation is the communication of the meaning of a source-languagetext by means of an equivalent target-language text. Translation beganonly after the appearance of written literature; there exist partialtranslations of the Sumerian Epic of Gilgamesh (ca. 2000 BCE) intoSouthwest Asian languages of the second millennium BCE.

As is apparent to anyone who has read a translation from a sourcenatural language (e.g., English, Spanish, etc.) into a destinationnatural language, it is very difficult, even for multi-lingual humans,to produce a translation that is faithful in meaning to the originalsource text. Computer-generated translations typically fall short ofthose produced by skilled humans, as evidenced by the fact that texttranslated from one language to another by a computer, and thentranslated back to the original language by the computer, often differsgreatly in meaning from the original text. A well-known example from theearly days of computer-based translation is a translation of the Englishsentence “The spirit is willing, but the flesh is weak” into Russian,and then back from Russian to English, with the result in English being“The vodka is good, but the meat is rotten.” Another well-known exampleillustrating the difficulties of translation is the English sentence“Time flies like an arrow, but fruit flies like a banana.”

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure is illustrated by way of example, and not by wayof limitation, and can be more fully understood with reference to thefollowing detailed description when considered in connection with thefigures in which:

FIG. 1 depicts an exemplary system architecture, in accordance with anembodiment of the present invention.

FIG. 2 depicts a block diagram of one embodiment of a translationevaluator.

FIG. 3 depicts a flow diagram of one embodiment of a first method forevaluating the quality of a translation.

FIG. 4 depicts a flow diagram of one embodiment of a second method forevaluating the quality of a translation from a first text in a firstnatural language to a second text in a second natural language.

FIG. 5 depicts a block diagram of an illustrative computer systemoperating in accordance with embodiments of the invention.

DETAILED DESCRIPTION

Described herein is a system and method for assessing the quality of atranslation from one natural language (e.g., English, Spanish, etc.) toanother natural language. In accordance with one embodiment, fourcategories are defined to assess translation quality: semantic quality,syntactic quality, stylistic quality, and formatting quality. Semanticquality is concerned with issues such as:

-   -   how faithfully the translation captures the meaning of the        original text;    -   the presence and quantity of extraneous content in the        translation;    -   content in the original text that is missing in the translation;        and    -   how closely the cultural context of the translation matches that        of the original text.

An example of the first issue is a translation of the sentence “Thewhite dishwasher is a Kenmore” that semantically interprets “whitedishwasher” incorrectly as a Caucasian person who washes dishes, ratherthan a dishwashing machine that is white in color. An example of thesecond issue is a translation of the English sentence “Barack Obama isin San Jose today” to a foreign-language equivalent of the Englishsentence “U.S. President Barack Obama left Washington D.C. today for SanJose, Calif.,” which is unnecessarily verbose. An example of the thirdissue is a translation of the English sentence “I saw Steven Spielbergat the premiere of ‘Lincoln’ in L.A.” to a foreign-language equivalentof the English sentence “I saw Steven Spielberg's movie ‘Lincoln’ when Iwas in L.A.” An example of the fourth issue is a first translation ofthe French sentence “J'aime le football” into “I like football” in theLondon Times and a second translation into “I like soccer” in the NewYork Times.

Syntactic quality is concerned with issues such as grammar, spelling,punctuation, usage of plural forms, tags, equations, and so forth.Stylistic quality is concerned with issues such as abbreviations andacronyms, capitalization, usage of appropriate examples and idioms,unity, readability, and so forth. Formatting quality is concerned withissues such as headers, footers, numbers, cross-references, fonts, whitespace, string length, and so forth.

In accordance with one example, translation quality is assessed bydetermining the number of errors in each category, as well as theseverity of the errors, and computing an overall translation qualityscore based on a weighted sum of the error counts. In another example, aclassifier is trained on a set of examples, each of which is aninput-output pair, where the input comprises a feature vector that isbased on the original text and the translated text, and the outputcomprises a subjective translation quality score that is provided by auser. The trained classifier is then used to assess the quality of aparticular translation by extracting features from the original text andtranslated text (e.g., word frequency, occurrence of word pairs, averagesentence length, etc.), generating a feature vector based on thefeatures, and providing the feature vector to the trained classifier toobtain an overall translation quality score.

In some examples, the errors are determined by one or more linguisticanalysis programs (e.g., a parser to determine grammatical errors, asemantic interpreter to determine semantic errors, etc.), while in someother examples errors associated with a particular category (e.g.,semantic, syntactic, formatting, etc.) are estimated by a classifier forthat category (e.g., a semantic error classifier that is trained oninput-output pairs mapping text to a semantic error count, a syntacticerror classifier that is trained on input-output pairs mapping text to asyntactic error count, etc.), while in yet other examples errors may bedetermined by a user.

In one embodiment, the quality of the translated text (or “target text”)is assessed in accordance with the following criteria:

-   -   target text should be faithful to the contextual meaning of the        source text;    -   target text should not have any additional or missing content        unless that is not required to understand the context and        meaning of translation;    -   headers, footers, numbers & cross-references should match with        the source text;    -   target text should use correct, popular, contemporary and well        accepted conventions of language grammar;    -   target text should not contain no spelling errors or typos and        should follow correct, popular, contemporary and well accepted        conventions of writing words;    -   target text should follow the appropriate convention of        capitalization of that language;    -   target text should follow the punctuation rules of the target        text;    -   target text should contain language use related to particular        domain or social setting of the audience of the target text;    -   target text should not contain any other language related        errors;    -   tag, syntax, equation, arguments should be intact in the target        text;    -   target text should use appropriate accelerators as per the        convention of language and community;    -   target text should place white-space as per the rule and        convention of the target language;    -   target text should use correct plural forms;    -   target text should be keyed in appropriate fonts either        suggested by the clients or required for the end product;    -   target text should not go beyond the prescribed string length;    -   target text should address the cultural context appropriately;    -   target text should present the country related locale data        correctly;    -   target text should contain and follow the language specific data        in present in locale;    -   target text should contain examples, idioms etc. according to        audience;    -   target text should follow the style guide if available and if        not available, the translation should follow the popular style        of the translation present in that domain;    -   target text should have high degree of internal unity and        readability;    -   target text should or should not use dialects of the target        language as per the guidelines or audience;    -   target text should contain the glossaries and terminology        prescribed or in case of glossary unavailability target text        should contains words and terms from popular convention of the        target language and related specialization;    -   target text should not contain inconsistent terms or words;    -   target text should contain abbreviation and acronym according to        convention of the language.

Embodiments of the present disclosure thus provide a mechanism by whichthe quality of a translation can be assessed in a quantitative fashion.The translation assessment can be applied to translations performed byhumans, as well as to translations performed by a computer. By assessingmultiple translations performed by the same person (or the same computerprogram) in this fashion, the translation skill of the person (orcomputer program) can be automatically evaluated and compared to otherpersons or computer programs.

FIG. 1 depicts an exemplary system architecture 100, in accordance withan embodiment of the present invention. The system architecture 100includes a server machine 115 and client machines 102A-102N connectedvia a network 104. The client machines 102A-102N may be personalcomputers (PC), laptops, mobile phones, tablet computers, or any othercomputing devices. The client machines 102A-102N may run an operatingsystem (OS) that manages hardware and software of the client machines102A-102N, and a browser that can access web pages of a web server andweb pages of a web-based application served by an application server.Network 104 may be a public network (e.g., the Internet), a privatenetwork (e.g., a local area network (LAN) or wide area network (WAN)),or a combination thereof.

Server machine 115 may be a rackmount server, a router computer, apersonal computer, a portable digital assistant, a mobile phone, alaptop computer, a tablet computer, a camera, a video camera, a netbook,a desktop computer, a media center, or any combination of the above.Server machine 115 includes an application server 140 and a translationevaluator 125. In alternative embodiments, the application server 140and translation evaluator 125 may run on different machines.

Application server 140 hosts a software application that receives textand translations of text from clients 102A-102N (e.g., text documentsthat are uploaded by clients 102A-102N via a Web-based graphicalinterface, etc.), provides the text and translations to translationevaluator 125, and that provides results from translation evaluator 125to clients 102A-102N (e.g., via Web pages, via summary reports that canbe downloaded from server machine 115 by clients 102A-102N, etc.).Translation evaluator 125 processes original and translated text andprovides assessments of the quality of the translated text (e.g., ascore that indicates how faithfully the translated text corresponds tothe original text, etc.). An embodiment of translation evaluator 125 isdescribed in detail below and with respect to FIGS. 2 through 4.

FIG. 2 is a block diagram of one embodiment of a translation evaluator200. The translation evaluator 200 may be the same as the translationevaluator 125 of FIG. 1 and may include a semantic analyzer 202, asyntactic analyzer 204, a style analyzer 206, an formatting analyzer208, a translation quality evaluator 210, and a data store 212.

The data store 212 may be a temporary buffer or a permanent data storeto hold source text, translated text, error counts, feature values,feature vectors, translation quality scores, and so forth.Alternatively, data store 212 may be hosted by one or more storagedevices, such as main memory, magnetic or optical storage based disks,tapes or hard drives, NAS, SAN, and so forth.

The semantic analyzer 202 determines one or more semantic-related valuesassociated with an original text and a translation of the original text,such as the number of errors in meaning in the translation as comparedto the original text, the number of errors in contextual context in thetranslation as compared to the original text, extraneous text in thetranslation, missing content in the translation, semantic features ofthe original text and the translation (e.g., the presence and frequencyof various semantic entities (people, places, activities, etc.) in theoriginal text and the translation, etc.), and so forth. In oneembodiment, the one or more semantic-related values are determined by aclassifier that has been trained on mappings between a vector ofsemantic features and corresponding semantic-related values. Someoperations of semantic analyzer 202 are described in more detail belowwith respect to FIGS. 3 and 4.

The syntactic analyzer 204 determines one or more syntactic-relatedvalues associated with an original text and a translation of theoriginal text, such as the number of grammatical errors in thetranslation as compared to the original text, the number of spellingerrors in the translation as compared to the original text, syntacticfeatures of the original text and the translation (e.g., the presenceand frequency of various syntactic entities (particular noun phrases,particular verb phrases, particular adjective phrases, etc.) in theoriginal text and the translation, etc.), and so forth. In oneembodiment, the one or more syntactic-related values are determined by aclassifier that has been trained on mappings between a vector ofsyntactic features and corresponding syntactic-related values. Someoperations of syntactic analyzer 204 are described in more detail belowwith respect to FIGS. 3 and 4.

The style analyzer 206 determines one or more stylistic-related valuesassociated with an original text and a translation of the original text,such as the number of abbreviation errors in the translation as comparedto the original text, the number of capitalization errors in thetranslation as compared to the original text, stylistic features of theoriginal text and the translation (e.g., the presence and frequency ofvarious stylistic-related entities (particular abbreviations, particularacronyms, particular idioms, etc.) in the original text and thetranslation, etc.), and so forth. In one embodiment, the one or morestylistic-related values are determined by a classifier that has beentrained on mappings between a vector of stylistic features andcorresponding stylistic-related values. Some operations of stylisticanalyzer 206 are described in more detail below with respect to FIGS. 3and 4.

The formatting analyzer 208 determines one or more formatting-relatedvalues associated with an original text and a translation of theoriginal text, such as the number of errors in usage of headers andfooters in the translation as compared to the original text, the numberof errors in usage of white space in the translation as compared to theoriginal text, formatting features of the original text and thetranslation (e.g., the presence, positioning, type, and size of variousformatting-related entities (e.g., headers and footers, fonts, whitespace, etc.) in the translation as compared to the original text, etc.),and so forth. In one embodiment, the one or more formatting-relatedvalues are determined by a classifier that has been trained on mappingsbetween a vector of formatting features and correspondingformatting-related values. Some operations of formatting analyzer 208are described in more detail below with respect to FIGS. 3 and 4.

The translation quality evaluator 210 determines an overall translationquality score for a translation based on semantic, syntactic, stylisticand formatting values. In one example, the overall translation qualityscore is calculated as a function of the semantic, syntactic, stylisticand formatting values, while in another example, a trained classifier isemployed to determine the overall translation quality score. Someoperations of translation quality evaluator 210 are described in moredetail below with respect to FIGS. 3 and 4.

FIG. 3 depicts a flow diagram of one embodiment of a method 300 forevaluating the quality of a translation from a first text in a firstnatural language to a second text in a second natural language. Themethod is performed by processing logic that may comprise hardware(circuitry, dedicated logic, etc.), software (such as is run on ageneral purpose computer system or a dedicated machine), or acombination of both. In one embodiment, the method is performed by theserver machine 115 of FIG. 1, while in some other embodiments, some orall of the method might be performed by another machine. It should benoted that blocks depicted in FIG. 3 can be performed simultaneously orin a different order than that depicted.

At block 301, one or more semantic-related values associated with atranslation of a first text into a second text are obtained. In oneexample, the one or more semantic-related values may include one or moreof the following:

-   -   one or more values based on how faithful (accurate) the meaning        of the second text is to the meaning of the first text (e.g.,        the number of minor semantic errors in the second text, such as        interpreting an Apple iPad as a smartphone rather than a tablet        computer; the number of major semantic errors in the second        text, such as interpreting “Robin” as a bird instead of a        person; the total number of semantic errors in the second text,        a weighted sum of the number of major and minor semantic errors        in the second text, etc.);    -   one or more values based on extraneous content in the second        text (e.g., the number of extraneous nouns or noun phrases in        the second text, such as referring to a “software program        application” rather than a “software application” or simply        “application;” the number of extraneous verb phrases in the        second text; the number of extraneous adjective phrases in the        second text; etc.);    -   one or more values based on missing content in the second text        (e.g., the number of missing noun phrases in the second text,        the number of missing verb phrases in the second text, the        number of missing adjective phrases in the second text, etc.);        and    -   one or more values based on how closely the cultural context of        the second text matches that of the first text (e.g., the number        of minor cultural context errors in the second text, the number        of major cultural context errors in the second text, the total        number of cultural context errors in the second text, a weighted        sum of the number of major and minor cultural context errors in        the second text, etc.).

It should be noted that in some other examples, the semantic-relatedvalues may include other items, such as errors related to language usewith respect to a particular domain or social setting of the audience ofthe translation, errors related inconsistent words or terms, and soforth, in addition to, or instead of, those enumerated above. It shouldfurther be noted that in some other examples, errors may be classifiedinto more than two categories, rather than just major and minor, and thevalues may also include one or more quality ratings (e.g, an integer ona scale from 1 to 10, etc.) for each of the subcategories enumeratedabove, rather than, or in addition to, the counting of errors. In oneembodiment, block 301 is performed by semantic analyzer 202.

At block 302, one or more syntactic-related values associated with thetranslation are obtained. In one example, the one or moresyntactic-related values may include one or more of the following:

-   -   one or more values based on grammatical usage (e.g., the number        of minor grammatical errors in the second text, the number of        major grammatical errors in the second text, the total number of        grammatical errors in the second text, a weighted sum of the        number of major and minor grammatical errors in the second text,        etc.);    -   one or more values based on spelling (e.g., the number of minor        spelling errors in the second text, the number of major spelling        errors in the second text, the total number of spelling errors        in the second text, a weighted sum of the number of major and        minor spelling errors in the second text, etc.);    -   one or more values based on punctuation (e.g., the number of        minor punctuation errors in the second text, the number of major        punctuation errors in the second text, the total number of        punctuation errors in the second text, a weighted sum of the        number of major and minor punctuation errors in the second text,        etc.);    -   one or more values based on usage of plural forms (e.g., the        number of minor plural form errors in the second text, the        number of major plural form errors in the second text, the total        number of plural form errors in the second text, a weighted sum        of the number of major and minor plural form errors in the        second text, etc.); and    -   one or more values based on non-text items such as equations and        tags (e.g., the number of minor errors in tags and equations in        the second text, the number of major errors in tags and        equations in the second text, the total number of errors in tags        and equations in the second text, a weighted sum of the number        of major and minor errors in tags and equations in the second        text, etc.).

It should be noted that in some other examples, the syntactic-relatedvalues may include other items in addition to, or instead of, thoseenumerated above, and that the errors may be classified into more thantwo categories, rather than just major and minor (e.g., four categories:trivial, venial, critical, blocker). It should further be noted that insome other examples, the syntactic-related values may also include oneor more quality ratings (e.g, an integer on a scale from 1 to 10, etc.)for each of the subcategories enumerated above, rather than, or inaddition to, the counting of errors. In one embodiment, block 301 isperformed by syntactic analyzer 204.

At block 303, one or more stylistic-related values associated with thetranslation are obtained. In one example, the one or morestylistic-related values may include one or more of the following:

-   -   one or more values based on abbreviation and acronym usage        (e.g., the number of minor abbreviation and acronym errors in        the second text, the number of major abbreviation and acronym        errors in the second text, the total number of abbreviation and        acronym errors in the second text, a weighted sum of the number        of major and minor abbreviation and acronym errors in the second        text, etc.);    -   one or more values based on capitalization (e.g., the number of        minor capitalization errors in the second text, the number of        major capitalization errors in the second text, the total number        of capitalization errors in the second text, a weighted sum of        the number of major and minor capitalization errors in the        second text, etc.); and    -   one or more values based on the appropriateness of examples and        idioms (e.g., the number of somewhat-inappropriate examples and        idioms in the second text, the number of very-inappropriate        examples and idioms in the second text, the total number of        inappropriate examples and idioms in the second text, a weighted        sum of the number of inappropriate examples and idioms in the        second text, etc.).

It should be noted that in some other examples, the stylistic-relatedvalues may include other items (e.g., unity, readability, dialect andlocale correctness, etc.) in addition to, or instead of, thoseenumerated above. It should further be noted that in some otherexamples, errors may be classified into more than two categories, ratherthan just major and minor, and the stylistic values may also include oneor more quality ratings (e.g, an integer on a scale from 1 to 10, etc.)for each of the subcategories enumerated above, rather than, or inaddition to, the counting of errors. In one embodiment, block 303 isperformed by style analyzer 206.

At block 304, one or more formatting-related values associated with thetranslation are obtained. In one example, the one or moreformatting-related values may include one or more of the following:

-   -   one or more values based on header and footer usage (e.g., the        number of minor header and footer errors in the second text, the        number of major header and footer errors in the second text, the        total number of header and footer errors in the second text, a        weighted sum of the number of major and minor header and footer        errors in the second text, etc.);    -   one or more values based on numbers and cross-references (e.g.,        the number of minor number/cross-reference errors in the second        text, the number of major number/cross-reference errors in the        second text, the total number of number/cross-reference errors        in the second text, a weighted sum of the number of major and        minor number/cross-reference errors in the second text, etc.);        and    -   one or more values based on font usage (e.g., the number of        minor font usage errors in the second text, the number of major        font usage errors in the second text, the total number of font        usage errors in the second text, a weighted sum of the number of        major and minor font usage errors in the second text, etc.); and    -   one or more values based on white space usage (e.g., the number        of minor white space usage errors in the second text, the number        of major white space usage errors in the second text, the total        number of white space usage errors in the second text, a        weighted sum of the number of major and minor white space usage        errors in the second text, etc.).

It should be noted that in some other examples, the formatting-relatedvalues may include other items (e.g., maximum string length,indentation, etc.) in addition to, or instead of, those enumeratedabove. It should further be noted that in some other examples, errorsmay be classified into more than two categories, rather than just majorand minor, and the stylistic values may also include one or more qualityratings (e.g, an integer on a scale from 1 to 10, etc.) for each of thesubcategories enumerated above, rather than, or in addition to, thecounting of errors. In one embodiment, block 304 is performed byformatting analyzer 208.

At block 305, an overall translation quality score is obtained based onthe semantic-related value(s) obtained at block 301, thesyntactic-related value(s) obtained at block 302, the stylistic-relatedvalue(s) obtained at block 303, and the formatting-related value(s)obtained at block 304. In one example, the overall translation qualityscore is obtained by calculating a normalized score for each of the fourcategories, and then computing a weighted sum of the four normalizedscores (e.g., weighing semantic quality more highly than formattingquality, etc.). In one embodiment, block 305 is performed by translationquality evaluator 210.

FIG. 4 depicts a flow diagram of one embodiment of a second method 400for evaluating the quality of a translation from a first text in a firstnatural language to a second text in a second natural language. Themethod is performed by processing logic that may comprise hardware(circuitry, dedicated logic, etc.), software (such as is run on ageneral purpose computer system or a dedicated machine), or acombination of both. In one embodiment, the method is performed by theserver machine 115 of FIG. 1, while in some other embodiments, some orall of the method might be performed by another machine. It should benoted that blocks depicted in FIG. 4 can be performed simultaneously orin a different order than that depicted.

At block 401, one or more semantic-related features associated with atranslation from a first text to a second text are obtained. In oneembodiment, the features may include the presence and frequency ofvarious semantic entities (e.g., people, places, activities, etc.) inthe first text and the second text, the presence and frequency ofvarious semantic relationships between entities (e.g., marriage,partnership, semantic actor/actee, etc.) in the first text and thesecond text, a cultural context identifier associated with the firsttext and with the second text, and so forth. In some examples, thesemantic entities may be identified by one or more linguistic analysisprograms (e.g., a parser to identify noun phrases, a semanticinterpreter to map noun phrases and verb phrases to semantic entities,etc.), while in some other examples the semantic entities may beidentified by a classifier (e.g., a classifier that has been trained oninput-output pairs mapping a semantic feature vector associated withtext to a set of semantic entities, etc.), while in yet other examplesthe semantic-related features may be determined by a user. In oneembodiment, block 401 is performed by semantic analyzer 202.

At block 402, one or more syntactic-related features associated with atranslation from a first text to a second text are obtained. In oneembodiment, the features may include the presence and frequency ofvarious syntactic entities (e.g., particular noun phrases, particularverb phrases, particular adjective phrases, etc.) in the first text andthe second text, syntactic relationships between entities (e.g.,subject/object, etc.) in the first text and the second text, and soforth. In some examples, the syntactic entities may be identified by oneor more linguistic analysis programs (e.g., a parser to identify partsof speech, etc.), while in some other examples the syntactic entitiesmay be identified by a classifier (e.g., a classifier that has beentrained on input-output pairs mapping a syntactic feature vectorassociated with text to a set of syntactic entities, etc.), while in yetother examples the syntactic-related features may be determined by auser. In one embodiment, block 302 is performed by syntactic analyzer204.

At block 403, one or more stylistic-related features associated with atranslation from a first text to a second text are obtained. In oneembodiment, the features may include the presence and frequency ofvarious stylistic-related entities (e.g., particular abbreviations,particular acronyms, particular idioms, etc.) in the first text and thesecond text, etc. In some examples, the stylistic entities may beidentified by one or more linguistic analysis programs (e.g., a parser,etc.), while in some other examples the stylistic entities may beidentified by a classifier (e.g., a classifier that has been trained oninput-output pairs mapping a stylistic feature vector associated withtext to a set of stylistic entities, etc.), while in yet other examplesthe stylistic-related features may be determined by a user. In oneembodiment, block 403 is performed by style analyzer 206.

At block 404, one or more formatting-related features associated with atranslation from a first text to a second text are obtained. In oneembodiment, the features may include the presence, positioning, type,and size of various formatting-related entities (e.g., headers andfooters, fonts, white space, etc.) in the first text and the secondtext, etc. In some examples, the formatting entities may be identifiedby one or more text layout or formatting programs (e.g., a desktoppublishing application, a layout editor, an image-processingapplication, a word processor, etc.), while in some other examples theformatting entities may be identified by a classifier (e.g., aclassifier that has been trained on input-output pairs mapping aformatting feature vector associated with text to a set of formattingentities, etc.), while in yet other examples the formatting-relatedfeatures may be determined by a user. In one embodiment, block 404 isperformed by formatting analyzer 208.

At block 405, a feature vector is generated based on thesemantic-related feature(s) obtained at block 401, the syntactic-relatedfeature(s) obtained at block 402, the stylistic-related feature(s)obtained at block 403, and the formatting-related feature(s) obtained atblock 404. In some examples, the feature vector may simply be acollection of the features obtained at blocks 401 through 404, while insome other examples, additional processing may be applied to thefeatures (e.g., a weighted sum, a normalized average, some othercombining function or mechanism, etc.) in the generation of the featurevector. In one embodiment, block 405 is performed by translation qualityevaluator 210.

At block 406, the feature vector generated at block 405 is provided to aclassifier to obtain an overall translation quality score. In oneembodiment, the classifier has previously been trained on a set ofexamples, each of which is an input-output pair, where the inputcomprises a feature vector based on an original text and a translationof the original text, and the output comprises a subjective translationquality score that is provided by a user. In one embodiment, block 406is performed by translation quality evaluator 210.

FIG. 5 illustrates an exemplary computer system within which a set ofinstructions, for causing the machine to perform any one or more of themethodologies discussed herein, may be executed. In alternativeembodiments, the machine may be connected (e.g., networked) to othermachines in a LAN, an intranet, an extranet, or the Internet. Themachine may operate in the capacity of a server machine in client-servernetwork environment. The machine may be a personal computer (PC), aset-top box (STB), a server, a network router, switch or bridge, or anymachine capable of executing a set of instructions (sequential orotherwise) that specify actions to be taken by that machine. Further,while only a single machine is illustrated, the term “machine” shallalso be taken to include any collection of machines that individually orjointly execute a set (or multiple sets) of instructions to perform anyone or more of the methodologies discussed herein.

The exemplary computer system 500 includes a processing system(processor) 502, a main memory 504 (e.g., read-only memory (ROM), flashmemory, dynamic random access memory (DRAM) such as synchronous DRAM(SDRAM)), a static memory 506 (e.g., flash memory, static random accessmemory (SRAM)), and a data storage device 516, which communicate witheach other via a bus 506.

Processor 502 represents one or more general-purpose processing devicessuch as a microprocessor, central processing unit, or the like. Moreparticularly, the processor 502 may be a complex instruction setcomputing (CISC) microprocessor, reduced instruction set computing(RISC) microprocessor, very long instruction word (VLIW) microprocessor,or a processor implementing other instruction sets or processorsimplementing a combination of instruction sets. The processor 502 mayalso be one or more special-purpose processing devices such as anapplication specific integrated circuit (ASIC), a field programmablegate array (FPGA), a digital signal processor (DSP), network processor,or the like. The processor 502 is configured to execute instructions 526for performing the operations and steps discussed herein.

The computer system 500 may further include a network interface device522. The computer system 500 also may include a video display unit 510(e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), analphanumeric input device 512 (e.g., a keyboard), a cursor controldevice 514 (e.g., a mouse), and a signal generation device 520 (e.g., aspeaker).

The data storage device 516 may include a computer-readable medium 524on which is stored one or more sets of instructions 526 (e.g.,instructions corresponding to the methods of FIGS. 2 through 5, etc.)embodying any one or more of the methodologies or functions describedherein. Instructions 526 may also reside, completely or at leastpartially, within the main memory 504 and/or within the processor 502during execution thereof by the computer system 500, the main memory 504and the processor 502 also constituting computer-readable media.Instructions 526 may further be transmitted or received over a networkvia the network interface device 522.

While the computer-readable storage medium 524 is shown in an exemplaryembodiment to be a single medium, the term “computer-readable storagemedium” should be taken to include a single medium or multiple media(e.g., a centralized or distributed database, and/or associated cachesand servers) that store the one or more sets of instructions. The term“computer-readable storage medium” shall also be taken to include anymedium that is capable of storing, encoding or carrying a set ofinstructions for execution by the machine and that cause the machine toperform any one or more of the methodologies of the present invention.The term “computer-readable storage medium” shall accordingly be takento include, but not be limited to, solid-state memories, optical media,and magnetic media.

Although the operations of the methods herein are shown and described ina particular order, the order of the operations of each method may bealtered so that certain operations may be performed in an inverse orderor so that certain operation may be performed, at least in part,concurrently with other operations. In another example, instructions orsub-operations of distinct operations may be in an intermittent and/oralternating manner.

In the foregoing description, numerous details have been set forth. Itwill be apparent, however, to one skilled in the art, that embodimentsof the present disclosure may be practiced without these specificdetails. In some instances, well-known structures and devices are shownin block diagram form, rather than in detail, in order to avoidobscuring the present disclosure.

Some portions of the detailed descriptions are presented in terms ofalgorithms and symbolic representations of operations on data bitswithin a computer memory. These algorithmic descriptions andrepresentations are the means used by those skilled in the dataprocessing arts to most effectively convey the substance of their workto others skilled in the art. An algorithm is here, and generally,conceived to be a self-consistent sequence of steps leading to a desiredresult. The steps are those requiring physical manipulations of physicalquantities. Usually, though not necessarily, these quantities take theform of electrical or magnetic signals capable of being stored,transferred, combined, compared, and otherwise manipulated. It hasproven convenient at times, principally for reasons of common usage, torefer to these signals as bits, values, elements, symbols, characters,terms, numbers, or the like.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise, as apparent from the foregoingdiscussion, it is appreciated that throughout the description,discussions utilizing terms such as “receiving”, “determining”,“executing”, or the like, refer to the action and processes of acomputer system, or similar electronic computing device, thatmanipulates and transforms data represented as physical (electronic)quantities within the computer system's registers and memories intoother data similarly represented as physical quantities within thecomputer system memories or registers or other such information storage,transmission or display devices.

The algorithms and displays presented herein are not inherently relatedto any particular computer or other apparatus. Various general purposesystems may be used with programs in accordance with the teachingsherein, or it may prove convenient to construct more specializedapparatus to perform the required method steps. In addition, embodimentsof the present disclosure are not described with reference to anyparticular programming language. It will be appreciated that a varietyof programming languages may be used to implement the teachings of thedisclosure as described herein.

Such a computer program may be stored in a computer readable storagemedium, such as, but not limited to, any type of disk including floppydisks, optical disks, CD-ROMs, and magnetic-optical disks, read-onlymemories (ROMs), random access memories (RAMs), EPROMs, EEPROMs,magnetic or optical cards, or any type of media suitable for storingelectronic instructions, each coupled to a computer system bus.Embodiments of the present disclosure may be provided as a computerprogram product, or software, that may include a machine-readable mediumhaving stored thereon instructions, which may be used to program acomputer system (or other electronic devices) to perform a processaccording to the present disclosure. A machine-readable medium includesany mechanism for storing or transmitting information in a form readableby a machine (e.g., a computer). For example, a machine-readable (e.g.,computer-readable) medium includes a machine (e.g., a computer) readablestorage medium (e.g., read only memory (“ROM”), random access memory(“RAM”), magnetic disk storage media, optical storage media, flashmemory devices, etc.), a machine (e.g., computer) readable transmissionmedium (electrical, optical, acoustical or other form of propagatedsignals (e.g., carrier waves, infrared signals, digital signals, etc.)),etc.

It is to be understood that the above description is intended to beillustrative, and not restrictive. Many other embodiments will beapparent to those of skill in the art upon reading and understanding theabove description. The scope of the invention should, therefore, bedetermined with reference to the appended claims, along with the fullscope of equivalents to which such claims are entitled.

What is claimed is:
 1. A method comprising: receiving, by a computersystem, data associated with a translation from a first text in a firstnatural language to a second text in a second natural language, whereinthe data comprises: a first value that is based on one or more semanticerrors of the second text, a second value that is based on extraneouscontent in the second text, and a third value that is based on missingcontent in the second text; and determining by the computer system,based on the first value, the second value, and the third value, a scorethat quantifies translation quality of the second text with respect tothe first text.
 2. The method of claim 1 wherein the score is determinedby a classifier that is executed by the computer system, and wherein theclassifier is trained on a set of input-output mappings, and wherein aninput of an input-output mapping is based on the first value, the secondvalue, and the third value, and wherein an output of an input-outputmapping is based on a subjective translation quality score provided by auser.
 3. The method of claim 2 wherein at least one of the first value,the second value, or the third value are determined by a classifier thatis executed by the computer system, and wherein the classifier istrained on a set of input-output mappings, and wherein an input of aninput-output mapping is based on a feature vector comprising a value ofa feature of the first text and a value of a feature of the second text,and wherein an output of an input-output mapping comprises at least oneof the first value, the second value, or the third value.
 4. The methodof claim 1 wherein the data further comprises a fourth value that isbased on one or more grammatical errors of the second text, and whereinthe determining is further based on the fourth value.
 5. The method ofclaim 1 wherein the data further comprises a fourth value that is basedon one or more punctuation errors of the second text, and wherein thedetermining is further based on the fourth value.
 6. The method of claim1 wherein the data further comprises a fourth value that is based on oneor more cultural context errors of the second text, and wherein thedetermining is further based on the fourth value.
 7. The method of claim1 wherein the data further comprises a fourth value that is based on oneor more errors of plural form usage of the second text, and wherein thedetermining is further based on the fourth value.
 8. An apparatuscomprising: a memory to store a first text in a first natural languageand a second text that is a translation of the first text into a secondnatural language; and a processor to: receive data associated with thetranslation, wherein the data comprises: a first value that is based onone or more semantic errors of the second text, a second value that isbased on extraneous content in the second text, and a third value thatis based on missing content in the second text; and determine, based onthe first value, the second value, and the third value, a score thatquantifies translation quality of the second text with respect to thefirst text.
 9. The apparatus of claim 8 wherein the processor executes aclassifier to determine the score, and wherein the classifier is trainedon a set of input-output mappings, and wherein an input of aninput-output mapping is based on the first value, the second value, andthe third value, and wherein an output of an input-output mapping isbased on a subjective translation quality score provided by a user. 10.The apparatus of claim 8 wherein at least one of the first value, thesecond value, or the third value are determined by the processor. 11.The apparatus of claim 10 wherein the processor executes a classifier todetermine at least one of the first value, the second value, or thethird value, and wherein the classifier is trained on a set ofinput-output mappings, and wherein an input of an input-output mappingis based on a feature vector comprising a value of a feature of thefirst text and a value of a feature of the second text, and wherein anoutput of an input-output mapping comprises at least one of the firstvalue, the second value, or the third value.
 12. The apparatus of claim8 wherein the data further comprises a fourth value that is based on oneor more errors of plural form usage of the second text, and wherein thedetermining is further based on the fourth value.
 13. The method ofclaim 1 wherein the data further comprises a fourth value that is basedon one or more capitalization errors of the second text, and wherein thedetermining is further based on the fourth value.
 14. The method ofclaim 1 wherein the data further comprises a fourth value that is basedon one or more terminology errors of the second text, and wherein thedetermining is further based on the fourth value.
 15. A non-transitorycomputer readable storage medium, having instructions stored therein,which when executed, cause a computer system to perform operationscomprising: receiving, by the computer system, data associated with atranslation from a first text in a first natural language to a secondtext in a second natural language, wherein the data comprises: a firstvalue that is based on one or more semantic errors of the second text, asecond value that is based on extraneous content in the second text, anda third value that is based on missing content in the second text; anddetermining by the computer system, based on the first value, the secondvalue, and the third value, a score that quantifies translation qualityof the second text with respect to the first text.
 16. Thenon-transitory computer readable storage medium of claim 15 wherein thescore is determined by a classifier that is executed by the computersystem, and wherein the classifier is trained on a set of input-outputmappings, and wherein an input of an input-output mapping is based onthe first value, the second value, and the third value, and wherein anoutput of an input-output mapping is based on a subjective translationquality score provided by a user.
 17. The non-transitory computerreadable storage medium of claim 15 wherein at least one of the firstvalue, the second value, or the third value are determined by thecomputer system.
 18. The non-transitory computer readable storage mediumof claim 17 wherein at least one of the first value, the second value,or the third value are determined by a classifier that is executed bythe computer system, and wherein the classifier is trained on a set ofinput-output mappings, and wherein an input of an input-output mappingis based on a feature vector comprising a value of a feature of thefirst text and a value of a feature of the second text, and wherein anoutput of an input-output mapping comprises at least one of the firstvalue, the second value, or the third value.
 19. The non-transitorycomputer readable storage medium of claim 17 wherein the data furthercomprises a fourth value that is based on one or more anachronisticusages of the second text, and wherein the determining is further basedon the fourth value.
 20. The non-transitory computer readable storagemedium of claim 17 wherein the data further comprises a fourth valuethat is based on one or more locale-specific context errors of thesecond text, and wherein the determining is further based on the fourthvalue.